1. Technical Field
The present invention is directed to an improved data processing system. More specifically, the present invention is directed to an apparatus and method for efficient implementation of queue pairs and completion queues that contain references to work queue entries and completion queue entries, respectively.
2. Description of Related Art
InfiniBand (IB) provides a hardware message passing mechanism which can be used for Input/Output devices (I/O) and Interprocess Communications between general computing nodes (IPC). Consumers access IB message passing hardware by posting send/receive messages to send/receive work queues on an IB Channel Adapter (CA). The send/receive work queues (WQ) are assigned to a consumer as a Queue Pair (QP). Consumers retrieve the results of these messages from a Completion Queue (CQ) through IB send and receive work completions (WC). The source CA takes care of segmenting outbound messages and sending them to the destination. The destination CA takes care of reassembling inbound messages and placing them in the memory space designated by the destination""s consumer.
There are two CA types: Host CA and Target CA. The Host CA is used by general purpose computing nodes to access the IB fabric. Consumers use IB verbs to access Host CA functions. The software that interprets verbs and directly accesses the CA is known as the Channel Interface (CI).
An efficient mechanism is needed to pass work requests from the consumer to the CA hardware. In addition, a similar mechanism is needed for the CA hardware to pass work completions to the consumer. Therefore, it would be advantageous to provide such a method, apparatus, and program, to pass work requests to channel adapter hardware and to pass work completions from the channel adapter hardware to the consumer, along with several optimization techniques.
The present invention provides a distributed computing system having (host and I/O) end nodes, switches, routers, and links interconnecting these components. The end nodes use send and receive queue pairs to transmit and receive messages. The end nodes use completion queues to inform the end user when a message has been completely sent or received and whether an error occurred during the message transmission or reception process. The present invention describes a mechanism for implementing these queue pairs and completion queues in hardware. A method for controlling the transfer of work requests from the consumer to the CA hardware and work completions from the CA hardware to the consumer using head and tail pointers that reference circular buffers is also described. The QPs and CQs do not contain Work Queue Entries and Completion Queue Entries respectively, but instead contain references to these entries. This allows them to be efficient and constant in size, while the Work Queue Entries and Completion Queue Entries themselves can vary in size, for example to include a variable number of data segments. Additionally, several mechanisms are provided which can be used to improve the overall efficiency of this process under different memory configurations.